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(57) ABSTRACT 

A system and method for inverse texture mapping in which 
given a 3D model and several images from different 
viewpoints, a texture map is extracted for each planar 
surface in the 3D model. The system and method employs a 
unique weighted pyramid feathering scheme for blending 
multiple images to form the texture map, even where the 
images are taken from different viewpoints, at different 
scales, and with different exposures. This scheme also 
makes it possible to blend images- with cut-out regions 
which may be present due to occlusions or moving objects. 
It further advantageously employs weight maps to improve 
the quality of the blended image. 
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INVERSE TEXTURE MAPPING USING FIG 1 illustrates the two foregoing problems in inverse 

WEIGHTED PYRAMID BLENDING texture mapping. Assume it is desired to create a texture map 

10 of the side wall 12 of the building 14 using two images 
16, 18. However, the view of the side wall 12 captured in the 
BACKGROUND 5 first image 16 is partially occluded by a tree 20. Similarly, 
. . . . the view of the side wall 12 capture in the second image 18 
technical Held ^ parlially occ i uded by another tree 22. In order to obtain a 
The invention is related to inverse texture mapping, and texture map of the side wall 12, the part of the two images 
more particularly, to a system and process for inverse texture ^ ig depicting the trees must be "cut-out" prior to blend- 
mapping using weighted pyramid blending. 10 ^g, ^s discussed above, this can be accomplished using an 
2. Background Art alpha mask and setting those pixels associated with the trees 
Image -based modeling and rendering has become an to zero. One of the problems addressed by the present 
important and powerful method to simulate visually rich invention is how to blend images with these cut-out regions 
virtual environments. A traditional method to represent such ( ie *> with alpha masks). Another problem addressed by the 
an environment is to use 3D geometry with some associated 15 present invention is how to blend images having different 
texture maps. Many techniques for recovering the 3D geom- sampling rates. This is also illustrated in FIG. 1. As can be 
etry of the environment from one or more images have been seen the view of the side wall 12 captured in the first image 
developed in the photogrammetry, computer vision, and 16 is directly in front of the wall, whereas the view of the 
computer graphics communities. These include both auto- side wall captured in the second image 18 is off to the side 
matic techniques [Fau93] and interactive techniques 20 of the wall. This means that the first image will provide a 
[DTM96]. better sampling rate of the pixels associated with the side 
High quality texture maps are essential for simulating waU 12 than will the second image in the context of image 
visually rich natural environments. Panoramic mosaics and viewpoint. However, it can also be seen that the second 
environment maps [Che95, M1395, and SS97] are examples ™*& was a P?« d at a grater zoom setting than the 
of rich texture maps used with very simple geometric 25 first ' and 50 ma V show more detad 0-*- a m g her sampling 

models (cylinders or cubes). rate > of the side waU 12 m some areas * ^ lhe resultin S 

_ . . . . , t t , , blended image would be more accurate if the contributions 

Anumber of blending algorithms have been developed to of OT ndi ixds in the two ; 16 18 where 

extract texture maps and/or high quality mosaics from w ^ led tQ ^ h ^ (jmd d of 

multiple images. A general requirement for blending two 3Q K M weU) rfor tQ luting. The issue of how to 

images is to make the seams between them invisible. One ^ ^ Mendi coefficients to account for these differ- 

common technique is to use lmear or nonlinear cross- ences fa addressed ^ { ^ ^ j 

dissolving or weighted averaging [Pel81Sze961 However, ^ ^ c ^ fce maintained ^ (he b £ nded 

choosing a proper cross-dissolving width is a diflicult task. • 

Burt and Adelson [BA83b] use a Laplacian pyramid to blend „ r .' , , . , , « 

two images in different frequency bands with different 35 u II 15 noic * mat m me P^ding paragraphs, as well as in 

blending widths. They can therefore blend without blurring the .^mamder of this specification the description refers to 

by cross-dissolving in narrower neighborhoods in higher vanous 1 ^dividual publications identified by an alphanu- 

frequency bands, and in wider neighborhoods in lower menc designator contained within a pair of brackets. For 

frpmiPnrv hand* example,, such a reference may be identified by recitmg, 

irequency oanos. „ - r . 40 "reference [DTM96]" or simply "[DTM96]". Multiple ref- 

Laplacian pyramid blending works very well for sphmng ereQces ^ be identified b & ^ of bfackets containi 

two overlappmg images Coining two unages at the center- more than Qne desi t for fc [?DS4 mm 

fine without visible seams) and has been widely adopted BH94b] A ^ Qf * e blications to each 

(e.g., for blending facial masks wi h face images [BCS97]). desi ^ can 5e found at me end of the Detailed Descrip- 

However, it cannot be directly applied to the inverse texture 45 fon^^ou 
mapping problem because it is not designed for images with 

alpha channels (visibility masks). SUMMARY 

Images with alpha channels, typically referred to as Given a 3D model and several images from different 

alpha-masked images, are widely used in many applications viewpoints, an "optimal" texture map can be extracted for 

such as image compositing [PD84, Bli94a, Bli94b] and 50 each planar surface in the 3D model using an inverse texture 

blue-screen matting [SB96]. In the context of inverse texture mapping process. In particular, a unique weighted pyramid 

mapping, images having cutouts due to occlusions, moving feathering process is employed that extends the traditional 

objects, and irregular shapes from perspective projection, Laplacian pyramid blending algorithm to handle alpha 

must often be blended. Alpha-masking the images provides masked images and take advantage of weight maps. In this 

a way to handle such images with cutouts. Specifically, the 55 way, it is possible to blend images having cut-out regions, 

pixels in the cut-out regions of the images are given zero such as those caused by occlusions or moving objects in the 

alpha values. However, as eluded to above, current blending images. In addition, the use of a weight map associated with 

methods are not capable of blending alpha masked images. each input image to indicate how much each pixel should 

In addition, when images are taken from different view contribute to the final, blended texture map, makes it pos- 

points or at different scales (i.e., zoom), they will have very 60 sible to create seamless texture maps, 

different spatial samplings when projected into the texture Specifically, the present invention is embodied in a system 

map, and therefore contribute content at different spatial and process for inverse texture mapping that uses the 

frequencies. Images to be blended may also have different aforementioned unique weighted pyramid feathering 

exposures. Before such images can be blended, a decision scheme. First, at least two images of a 3D scene are inputted 

must be made as to how much each image (in fact, each pixel 65 and any pixel located in a cut-out region of the images is set 

in each image) should contribute to the final, blended texture to a zero value. The 3D scene is characterized as a collection 

map. of regions (e.g., triangles). All the regions of the 3D scene 
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which are to be texture mapped are identified in each In addition to the just described benefits, other advantages 

inputted image. A 2D perspective transform for each iden- of the present invention will become apparent from the 

tified planar region is then computed. This transform is detailed description which follows hereinafter when taken in 

capable of projecting the associated planar region to pre- conjunction with the drawing figures which accompany it. 

scribed texture map coordinates. The 2D perspective trans- 5 

forms are used to warp each identified planar region to the DESCRIPTION OF THE DRAWINGS 
prescribed texture map coordinates to create plurality of 

warped images. A weight map is created for each warped The file of this patent contains at least one drawing 

image. As indicated above, the weight maps specify the executed in color. Copies of this patent with color 

degree to which each pixel in a particular warped image is drawing(s) will be provided by the Patent and Trademark 

to contribute to the final, blended image. Office upon request and payment of the necessary fee. 

The next step in the overall inverse texture mapping The specific features, aspects, and advantages of the 

process is to blend the warped images using the weight present invention will become better understood with regard 

maps. This generally entails forming an alpha premultiplied to the following description, appended claims, and accom- 

image from each warped image and constructing a band- 15 panying drawings where: 

pass Laplacian pyramid from the alpha premultiplied FIG. 1 is a diagram graphically depicting some of the 

images. In addition, a low-pass Gaussian pyramid is con- problems associated with creating a texture map of a side of 

structed from each of the previously created weight maps. A a building using two images of the building taken from 

new composite band-pass Laplacian pyramid is then com- different viewpoints, at different scales (i.e., zoom) and each 

puted using the band-pass Laplacian pyramids and low-pass 20 an intervening object causing an occlusion in the 

Gaussian pyramids associated with the warped images. This image. 

computation include*, for each corresponding level of the pic 2 is a diagram depicting a general purpose comput- 
pyramids, first multiplying each pixel of each warped image ^ device an exemplary system for implement- 
by the weight factor associated with that same pixel location • , he m ^^0^ 
in the corresponding weight map to produce a weighted ,,- „...,,. r 

pixel value. The weighted pixel values for each correspond- 9' 3 15 a block diagram ° f a ° 0VeraU P . rOCeSS for 

ing pixel location in all the warped images are respectively creatm S a texture ma P according to the present invention, 

added together and divided by the sum of the weight factors FIG - 4 k a diagram graphically depicting the construction 

corresponding with that pixel location. Lastly, the blended of Gaussian and Laplacian pyramids, 

image is produced by expanding and summing the pyramid 30 FIGS. 5A and 5B are block diagrams of a process for 

levels of the composite band-pass Laplacian pyramid, and accomplishing the weighted pyramid feathering blending 

compensating for the previously implemented alpha premul- _ module of the overall process of FIG. 3. 

" tiplication of the warped images. " ' " RGS 6A and 6fi are images depicting the results of the 

In regards to the alpha premultiplied, this process inverse texture mapping process of FIG. 3 shown before the 

involves multiplying the R, G and B components of each 35 performance of the final logical OR operator step in FIG. 6A 

warped image pixel by the alpha value associated with that and after the performance of the step in FIG. 6B. 

pixel. Thus, the preferred alpha premultiplication compen- FIGS 7A througn n are images depicting apples and 

sation step involves dividing the R,G,B components of each oranges, some with occluded regions, and the results of 

pixel of the blended image by its alpha value. using lhe - mv ^ texture mappmg process of FIG. 3 to blend 

It is also preferred that measures be taken to determine 40 the images, 

whether the warped images are in registration with one FIGS 8A throu ^ gD are ^ depicting a bookshelf 

another pnor to the blendmg process. If the images are ^ tQ mustrate me advant of the inverse texture 

mis-registered, the blended image quality can suffer. If any m ^ of nG 3 ovef conventional feathering 

unacceptable mis-registration is discovered the warped whefl ^ { bei MeDded Qave objects wMch 

images are aligned via conventional methods [SS97, SS98]. 45 have moved between lhe time each ^ was taken 

In addition, it is preferred that, just prior to compensating HGS 9A tfa 9D afe 

for the alpha premultiplication in the blended image, the R, „. A t * * «u * * * *u • 

r> ™ j 1 l . c . ■ j • £l I an office used to illustrate the advantages of the inverse 

G, B and alpha components of each pixel be cupped by first t _ . c rir , - & , 

r*u * u ■ 1 ri \. texture mappmg process of FIG. 3 over conventional aver- 

setting any of these components having a value of less than . , r v fu • u l *u • i_ • 

f J . *1 „. 6 f „ ~ j j-* agmg and feathering schemes when the images being 

zero to zero, and secondly setting any of the R, G, and B 50 . f 5 a j*<t ♦ 

. . 7 , * u * ■ . ,1 j . blended exhibit different exposures, 

components having a value greater than alpha to the alpha r 

value. This will compensate for any underflow or overflow DETAILED DESCRIPTION OF THE 

caused by the pyramid blending process. It is also desirable PREFERRED EMBODIMENTS 

that a step be added to the general blending process 

described previously to ensure any cut-out region common 55 In the following description of the preferred embodiments 

to all the input images maintains a zero alpha value. This is of the present invention, reference is made to the accompa- 

accomplished via the use a logical OR operator following nying drawings which form a part hereof, and in which is 

the alpha premultiplication compensation step. Specifically, shown by way of illustration specific embodiments in which 

each pixel of the blended image is compared to a corre- foe invention may be practiced. It is understood that other 

spondingly located pixel of each of the alpha premultiplied 60 embodiments may be utilized and structural changes may be 

images. Whenever any compared pixel of the blended image made without departing from the scope of the present 

coincides with a zero-valued pixel in all of the alpha invention. 

premultiplied images, it is set to zero no matter what value FIG. 2 and the following discussion are intended to 

it obtained during the blending process. Conversely, when- provide a brief, general description of a suitable computing 

ever any compared pixel of the blended image does not 65 environment in which the invention may be implemented, 

coincide with a zero-valued pixel in all of the alpha pre- Although not required, the invention will be described in the 

multiplied images, it is left unchanged. general context of computer-executable instructions, such as 
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program modules, being executed by a personal computer. 
Generally, program modules include routines, programs, 
objects, components, data structures, etc. that perform par- 
ticular tasks or implement particular abstract data types. 
Moreover, those skilled in the art will appreciate that the 
invention may be practiced with other computer system 
configurations, including hand-held devices, multiprocessor 
systems, microprocessor-based or programmable consumer 
electronics, network PCs, minicomputers, mainframe 
computers, and the like. The invention may also be practiced 
in distributed computing environments where tasks are 
performed by remote processing devices that are linked 
through a communications network. In a distributed com- 
puting environment, program modules may be located in 
both local and remote memory storage devices. 

With reference to FIG. 2, an exemplary system for imple- 
menting the invention includes a general purpose computing 
device in the form of a conventional personal computer 20, 
including a processing unit 21, a system memory 22, and a 
system bus 23 that couples various system components 
including the system memory to the processing unit 21. The 
system bus 23 may be any of several types of bus structures 
including a memory bus or memory controller, a peripheral 
bus, and a local bus using any of a variety of bus architec- 
tures. The system memory includes read only memory 
(ROM) 24 and random access memory (RAM) 25. A basic 
input/output system 26 (BIOS), containing the basic routine 
that helps to transfer mformation between elements within 
the personal computer 20, such as during start-up, is stored 
in ROM 24. The personal computer 20 further includes a 
hard disk drive 27 for reading from and writing to a hard 
disk, not shown, a magnetic disk drive 28 for reading from 
or writing to a removable magnetic disk 29, and an optical 
disk drive 30 for reading from or writing to a removable 
optical disk 31 such as a CD ROM or other optical media. 
Hie hard disk drive 27, magnetic disk drive 28, and optical 
disk drive 30 are connected to the system bus 23 by a hard 
disk drive interface 32, a magnetic disk drive interface 33, 
and an optical drive interface 34, respectively. The drives 
and their associated computer-readable media provide non- 
volatile storage of computer readable instructions, data 
structures, program modules and other data for the personal 
computer 20. Although the exemplary environment 
described herein employs a hard disk, a removable magnetic 
disk 29 and a removable optical disk 31, it should be 
appreciated by those skilled in the art that other types of 
computer readable media which can store data that is 
accessible by a computer, such as magnetic cassettes, flash 
memory cards, digital video disks, Bernoulli cartridges, 
random access memories (RAMs), read only memories 
(ROMs), and the like, may also be used in the exemplary 
operating environment. 

A number of program modules may be stored on the hard 
disk, magnetic disk 29, optical disk 31, ROM 24 or RAM 25, 
including an operating system 35, one or more application 
programs 36, other program modules 37, and program data 
38. A user may enter commands and information into the 
personal computer 20 through input devices such as a 
keyboard 40 and pointing device 42. Of particular signifi- 
cance to the present invention, a camera 55 (such as a 
digital/electronic still or video camera, or film/photographic 
scanner) capable of capturing a sequence of images 56 can 
also be included as an input device to the personal computer 
20. The images 56 are input into the computer 20 via an 
appropriate camera interface 57. This interface 57 is con- 
nected to the system bus 23, thereby allowing the images to 
be routed to and stored in the RAM 25, or one of the other 
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data storage devices associated with the computer 20. 
However, it is noted that image data can be input into the 
computer 20 from any of the aforementioned computer- 
readable media as well, without requiring the use of the 

5 camera 55. Other input devices (not shown) may include a 
microphone, joystick, game pad, satellite dish, scanner, or 
the like. These and other input devices are often connected 
to the processing unit 21 through a serial port interface 46 
that is coupled to the system bus, but may be connected by 

10 other interfaces, such as a parallel port, game port or a 
universal serial bus (USB). A monitor 47 or other type of 
display device is also connected to the system bus 23 via an 
interface, such as a video adapter 48. In addition to the 
monitor, personal computers typically include other periph- 

15 eral output devices (not shown), such as speakers and 
printers. 

The personal computer 20 may operate in a networked 
environment using logical connections to one or more 
remote computers, such as a remote computer 49. The 

20 remote computer 49 may be another personal computer, a 
server, a router, a network PC, a peer device or other 
common network node, and typically includes many or all of 
the elements described above relative to the personal com- 
puter 20, although only a memory storage device 50 has 

25 been illustrated in FIG. 2. The logical connections depicted 
in FIG. 2 include a local area network (LAN) 51 and a wide 
area network (WAN) 52. Such networking environments are 
commonplace in offices, enterprise-wide computer 
networks, intranets and the Internet. 

30 When used in a LAN networking environment, the per- 
sonal computer 20 is connected to the local network 51 
-through a network interface or adapter 53. When used in a 
WAN networking environment, the personal computer 20 
typically includes a modem 54 or other means for establish- 

35 ing communications over the wide area network 52, such as 
the Internet. The modem 54, which may be internal or 
external, is connected to the system bus 23 via the serial port 
interface 46. In a networked environment, program modules 
depicted relative to the personal computer 20, or portions 

40 thereof, may be stored in the remote memory storage device. 
It will be appreciated that the network connections shown 
are exemplary and other means of establishing a communi- 
cations link between the computers may be used. 

45 The exemplary operating environment having now been 
discussed, the remaining part of this description section will 
be devoted to a description of the program modules embody- 
ing the invention and the testing of these modules. 

The general approach to texture map recovery assumes 

50 that a 3D model has been reconstructed, preferably using 
automatic algorithms (e.g., structure from motion) or inter- 
active techniques (e.g., [SHS98]). Given the 3D model and 
multiple images (possibly taken from different viewpoints) 
and their associated world-to-camera transformations, tex- 

55 ture maps can be extracted that are blended seamlessly from 
all input images. This process is referred to as inverse texture 
mapping because the input images are projected (through 3D 
model) from camera views into the texture view, as opposed 
to (forward) texture mapping from texture maps (through the 

60 3D model) into the screen view (i.e., a new camera view). 
An overview of a preferred inverse texture mapping 
process embodying the present invention is depicted in FIG. 
3. The inverse texture mapping problem is decoupled into 
one of reconstructing texture maps for any or all planar 

65 surfaces (e.g., triangles) in which the 3D model has been 
divided. For every planar surface in the scene that is to be 
mapped, the first step 300 in the process is to compute a 2D 
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perspective transform capable of projecting the planar sur- 
face from image to texture map coordinates [SS97], The 
texture map coordinates (u, v) associated with the vertices of 
each planar surface are supplied ahead of time, and can 
either be regular 2D indices (u, v), or 3D homogeneous 5 
coordinates (u, v, w), in the case of projective texture maps 
[SKvW+92]. Each planar surface from each image is then 
warped via conventional methods using the previously com- 
puted 2D perspective transforms to form warped versions of 
the input images (step 302). 10 

Before the warped images can be blended together, it 
should be determined whether they are in registration with 
one another via conventional methods [SS98]. If an unac- 
ceptable amount of mis-registration exists, it will be neces- 
sary to do local alignment, or "deghosting" [SS98] prior to 15 
blending in order to obtain the desired results (step 304). 
Once any unacceptable mis-registration has been corrected 
, the warped images are blended using a weighted pyramid 
feathering scheme (step 306) that will be described in detail 
later in this description. However, first the weight maps that 20 
play an integral part in the blending scheme must be created. 
1. Creating Weight Maps 

When blending several images to create a texture map, the 
preferred course of action is to decide how much each image 
(or more particularly each pixel from each image) contrib- 25 
utes to the final texture map. This is accomplished by 
creating a weight map for each image which dictates on a 
pixel by pixel basis how much that pixel will contribute to 
the blended image. These weight maps are then used in the 
pyramid feathering algorithm as will be described later to 30 
create the texture maps. The use of the weight maps for each 
image facilitates their seamless blending regardless of the 
differences in sampling (e.g., different viewpoints or differ- 
ent scales) or exposure between the images. 

There are many existing weighting schemes that can be 35 
adopted for the purpose of producing an acceptable weight 
map for use with the weighted pyramid feathering process. 
Two of the more preferred of the existing methods will be 
presented herein. Namely the binary mask method and the 
"distance from image contour" method. However, it is not 40 
intended to limit the present invention to just embodiments 
employing these two methods. Rather, any appropriate exist- 
ing weighting scheme can be employed. For instance, a 
simple averaging method would be another example of a 
weighting process that could be employed. 45 

1.1 Binary Masks 

The simplest possible weight map is a binary mask that 
can be associated with each image to dictate which pixels 
should be used in the texture map and which should not. For 
each image, a pixel is marked if its distance to its own image 50 
center is closer than its distance to any other image center. 
Otherwise the pixel is not used in blending. This process is 
similar to constructing the Voronoi diagram of multiple 
image centers [W+97, PH97]. The binary mask method is 
particularly effective when images have significant overlaps. 55 

However, when image centers overlap, there are many 
other choices for the weight map, as well. For example, we 
can simply choose the left half from one image and the right 
half from the other image. It should be noted however that 
these binary masks cannot deal with cut-out regions, nor 60 
images taken at different scales. 

1.2 Distance from Image Contours 

A better heuristic is a weight map which measures the 
minimum distance of each pixel to the image edges or 
nearest zero-alpha pixels. In this scheme, the farther a pixel 65 
is from the image center, the less important it is. A city -block 
distance or grassfire transform [RK76] can be used to 
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establish the required distances in this method. Because 
these distance measures typically weight the center of the 
image more heavily than edges, it is called a "feather map" 
when applied to image mosaicing [Sze96, SS97]. 

The feather map can deal with cut-out regions by simply 
assigning zero weight to the zero alpha regions. However, a 
linear feather map (e.g., one employing city-block distance) 
has an undesirably wide transition zone. To narrow the 
transition zone, a non-linear feather map d x can be employed 
instead. In this scheme, the non-linear feather map d a is 
computed from the city-block distance d as 

di-^s 2 (1) 

where s-0,5*(1.0-cos(c//d majc )), and d max is the largest city- 
block distance. 

Compared with the Voronoi diagram approach, the non- 
linear feather map is easier to implement (e.g., no need to 
check for the Voronoi boundary) and has a more gradual 
transition. 

2. Blending with Weighted Pyramid Feathering 

2.1 Burt and Adelson's Pyramid Blending 

A pyramid blending technique capable of seamlessly 
blending two images was previously developed by P. J. Burt 
and E. H. Adelson [BA83b] (hereinafter referred to as the 
Burt and Adelson pyramid blending technique). In this 
technique, two input images are first decomposed into 
band-pass pyramids (i.e., Laplacian pyramids [BA83a]) 
which are blended at different levels and then summed back 
to the finest level to reconstruct a blended image. 

To blend two (completely overlapped) images A and B, 
according to a given mask, the Laplacian pyramid blending 
works in the following three steps: 

a. Construct Laplacian pyramids LA and LB for images A 
and B, respectively; 

b. Construct a new Laplacian pyramid LS by taking the 
weighted average of LA and LB at every pyramid level 
using the given mask; 

c. Reconstruct a new image S by summing all levels of 
LS. 

A mask needs to be specified at every pyramid level for 
proper blending. If the mask is only specified at the lowest 
level, its Gaussian pyramid can be used for the other 
pyramid levels. The Gaussian pyramid also makes the 
transition at the mask contour smoother as it tends to widen 
the transition zone at coarser levels of the pyramid. 

To construct the band pass pyramid for an image, the first 
step is to construct its low-pass pyramid (Gaussian 
pyramid). The Gaussian pyramid is obtained by a REDUCE 
operation which involves low-pass filtering (i.e., convolving 
with a small weighting kernel such as a vertical and a 
horizontal {1, 4,6,4,1 }/16 kernel for a 5x5 window) fol- 
lowed by downsampling. A Laplacian pyramid at level 1 is 
the difference between the Gaussian pyramid at level 1 and 
the expanded Gaussian pyramid at level 1 (i.e., expanded 
using an EXPAND operator from level 1+1 to 1). The 
aforementioned expansion process involve is upsampling 
followed by low-pass filtering. FIG. 4 illustrates the con- 
struction of the Gaussian and Laplacian pyramids. 

2.2 Pyramid Blending with Alpha 

When part of an input image is occluded, that part has to 
be cut out and should not be used to construct the texture 
map. Images with cut-out regions can be represented by 
alpha masks where alpha values inside cut-out regions are 
set to zero. 

To blend two images with alpha masks, it is preferable to 
use alpha premultiplied images (i.e., ccR, aG, aB, a), 
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instead of (R, G, B, a), as suggested by [PD84] and others. process of Laplacian pyramid reconstruction, as shown in 
As shown in Blinn's example of downsampling and over- FIG. 6. In operation, if corresponding pixels in all the 
laying [Bli94a], all filtering should be done on alpha pre- warped alpha-masked images are assigned a zero value, then 
multiplied (associated) colors rather than unmultiplied col- the same pixel in the blended image is also assigned a zero 
ors. The use of alpha premultiplied images in the present 5 value, even if the blending process has produced a pixel 
invention is advantageously employed to construct the value exceeding zero. In this way, any cut-out region com- 
aforementioned Laplacian pyramid since the construction mon t0 a11 the blended images is maintained. Conversely, if 
and reconstruction of Laplacian pyramid involves a combi- anv or ^ of tne corresponding pixels in the warped alpha- 
nation of filtering operations (REDUCE, EXPAND, masked images has a non-zero value, then the pixel value of 
subtraction, etc.). It will be shown that for input images with 10 the blended image is maintained as is. 
alpha maps, Laplacian pyramid blending can be easily and . There a ' e | w ° desired properties when we blend two 
correctly applied to all four channels (aR, aG, aB, a). ! ma Sf ™ lh * £ h ? raasks A F /f 1 < ha j ™jp AMendcd with A 
2 3 Wei hted Pyramid Featherin 15 a image A blended with another image B is 
^ •tj m_jt. jaji >vij* A, if B is A with a cut-out mask (zero alpha). The first 
The previously described Burt and Adelson s blending is me reconstructio > D 0 ' f me band . 

method can be modified to produce a pyramid feathering 15 pass pyramid ^ the same as the ori inal ^ ^ the 

process embodying the present invention. As illustrated in weights m normalized at ^ id levels> Unfortunately, 

FIGS. 5A and 5B, this new process includes the steps of: property ^ generaUy not satisfied cui<3Ui 

a) For each warped image k (which is comprised of all the contours. However, in practice, the reconstruction error of 
warped planar regions associated with the image): blending the cutout image B with its original image A, 

(i) create a weight map as described previously (step 20 usmg tne foregoing weighted pyramid feathering process, is 
500); 5Q sma u as to be insignificant. For instance, it has been 

(ii) create an alpha premultiplied image from the determined experimentally that a blend of the "rotten" 
warped image k as described previously (step 502); orange (p IG 7 q ^ lhe orange (pjQ 73^ or the « rotteri " 

(iii) construct a band-pass Laplacian pyramid L k from app i e (piG. 7G) with the apple (FIG. 7D) yields a RMS error 
the alpha premultiplied image (step 504); and 25 0 f 0,6 gray level (i.e., 0.3% relative to intensity value), 

(iv) construct a low-pass Gaussian pyramid w* from the which ^ visually indistinguishable, 
weight map (step 506), including setting w*=0 wher- 3 Tested Embodiments 

ever A computer program embodying many of the aspects of 

b) Compute a new composite band-pass Laplacian pyra- the present invention including the weighted pyramid feath- 
mid LS using each band-pass Laplacian pyramid Ljt and 30 eT \ D g process was coded in visual C ++ and run on a 
its associated low-pass Gaussian pyramid w* for each conventional PC. The program in its coded form is storable 
pyramid. level 1, via the following equation (step 508): 0 n. and can be executed from any appropriate storage 

medium, such as the hard disk, magnetic disk 29, optical 

£ (tf'f. » ■ *?'<'. /)) (2) disk 31 » R0M 24 or RAM 25 discussed in connection with 

Z^Of/, D = ' ^ , / indexes the pyramid level 35 FIG ' L ^ P rocesS was a PP lied to a number of images, 

2j "i'^ /) < some of which are detailed below. 

* By way of an example, the pyramid feathering algorithm 

was used to blend the image of an apple and an orange, both 

• T «m /• *\ • * * *u t n • c of which had cut-out zero alpha masks. For comparison, the 

where LS 1 } (1, t) is set to zero if the sum of all weights of ^ . . . t . r . . , A *_ . , t , , _ _ , r ' 

pixel (i i) at level 1 is zero left half of the apple ^ the nght half of ^ oran S e are 

.1 «t j- shown in FIG. 7A. Notice the distinct seam where the two 

c) Construct a new blended image S by expanding and halyes of ^ ^ meet ^ ^ ^ dq 

summing all the levels 1 of the composite band-pass ^ wd h(ed ^ fcatherin ^ duced the 

Laplacian pyramid LS (step 510). image shown in FIG. 7E. Notice that the transition between 

At the end of Laplacian reconstrucUon, there may be 45 ^ k and e ^ almos , mdistmguishable rela i ive to 
overflow and underflow associated with the R, G B, and/or mQ yA Tq obtain ^ strfldn ^ w( , used wei ^ t 

alpha values. There is no problem that negative alpha values c , A * « 

" , * . & t maps for the apple and the orange as follows, 
exist at different levels of a band-pass pyramid, but to be 

able to display the final reconstructed image, alpha should be if *$l/2 n if * 

limited to a number between 0 and 1. It is noted, however, 50 w appte (J) ~\ 1 ; and w cn ^ t (j) = l 1 ' 

that in practice this range of alpha numbers is typically ^ e othenvise l e olhe 
represented by 8-bit alpha values between 0 and 255. This 

limiting or clipping operation involves the following (step wnere e ^ a sma u num ber (for our examples, we have used 

512): 0.1) and L is the image width. 

a) If a<0, a«0; if R<0, R«0 ; if G<0, G=0; if B<0, B=0; 55 It is also possible to obtain a seamless blending, even with 
and the "rotten*' apple and orange images of FIGS. 7C and 7G, 

b) If R>ct, R»a; if G>a, G=a; if B>a, B=a. respectively, using the weighted pyramid feathering process. 
Once the clipping operation is complete, the next step 514 The term "rotten" refers to the fact that these images have 
is to "un-multiply" R, G, B using the a value (i.e., divide R, cut-out regions. As was stated earlier, our method is good at 
G, B by the a value associated with the pixel) 60 generating texture maps from multiple images even when 

As an optional final step 516, the new image can be parts of the images have been cut out. The results of our 

multiplied with the logical OR of all the original warped seamless blending process can be seen in FIG. 7F where the 

alpha masks, thereby trimming the composite to have non- image of the "rotten" orange (FIG. 7C) is blended with the 

zero pixels where these exist in all the original images. This image of the intact apple (i.e., 7D). Similarly, this seamless 

final step is necessary if the original alpha map (in particular 65 blending process is illustrated in FIG. 7H where the image 

zero alpha regions) needs to be preserved, because incorrect of the "rotten" apple (FIG. 7G) is blended with the image of 

colors can "bleed" into the zero alpha mask region in the the intact orange (i.e., FIG. 7B). 



a L/2 (3) 
otherwise 
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Note that our purpose here is not to fill the hole, nor to hybrid geometry- and image-based approach. Computer 

synthesize new texture, but to blend two images seamlessly Graphics (SIGGRAPH'96), pages 11-20, August 1996. 

using both the apple and orange images. Thus, as depicted [Fau93] 0. Faugeras. Three-dimensional computer vision: A 

in FIG. 71, the blending of the images of FIGS. 7C and 7G geometric viewpoint. MIT Press, Cambridge, Mass. 1993. 

results in a seamlessly blended image, but one exhibiting a 5 [GGSC96] S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. 

cut-out region. If we wished to fill the hole, we could use R Cohen. The lumigraph. In Computer Graphics 

another orange image (or part of the same orange image) and Proceedings, Annual Conference Series, pages 43-54, 

blend them together. Other conventional hole filling meth- Jf™™ 96 (New Orleans), August 1996. ACM 

ods that could be employed include the noise removal r^TioiVv^ j « Tf , L _ ^ ^ • 

*u j ruT^i a i * * mat u^ ,n IGH86J N. Greene and P. Heckbert. Creating raster Omm- 

method [HT96], pyramidal texture synthesis method 10 L J . l4 . . ? . 

rT>r»fV7i j L II • fin- #u a rrrcpn^ max images from multiple perspective views using the 

[E S £ USh ' P ^ IC &™- mQ Z m , e u [G ° SC96 t, eUiptical Weighted average filter. Computer Graph- 

FIGS. 8A-D provide another example where the capabil- ics and Applications, 6(6):21-27, June 1986. 

ity to seamlessly blend images having cut-out regions can be [ Hec 86] P. Heckbert. Survey of texture mapping. IEEE 

advantageous. FIG. 8A shows two images where a cup has Computer Graphics and Applications, 6(ll):56-67, 

been moved between the time each image was captured. A 15 November 1986. 

blended image obtained using a conventional feather blend- [HT96] A. N. Hirani and T. Totsuka. Combining frequency 

ing algorithm exhibits a ghost (or double image) of the cup, and spatial domain information for-fast interactive image 

as shown in FIG. 8B. To prevent this ghosting problem, the noise removal. Computer Graphics (SIGGRAPH96), 

cups can be "cut-out" of each image using an alpha masking pages 269-276, August 1996. 

process, as shown in FIG. 8C. Applying our weighted 20 [MB95] L. McMillan and G. Bishop. Plenoptic modeling: 

pyramid feathering process to the cut-out images in FIG. 8C An image-based rendering system. Computer Graphics 

leads to the seamlessly blended image in FIG. 8D. Notice (SIGGRAPH' 95), pages 39-46, August 1995. 

that the ghost image of the cup seen in FIG. 8B has been [PD84] T Porter and T. Duff. Compositing digital images, 

eliminated. Computer Graphics (SIGGRAPH'84), 18(3):253-259, 

Another key advantage of using the pyramid blending 25 July 1984. 

algorithm according to the present invention is that input [Pel81] S. Peleg. Elimination of seams from photomosaics. 

images having vastly different exposures can be seamlessly In Proceedings of the Conference on Pattern Recognition 

blended. FIG. 9A shows two images of the same scene, but and Image Processing, pages 426-429, Dallas, Tex., 

with different exposures. FIG. 9B shows the result of using August 1981. 

a simple averaging blending method, and FIG. 9C shows. 30 [PH97] S. Peleg and J. Herman. Panoramic mosaics by 

the result of using a conventional feathering blending manifold projection. In IEEE Computer Society Confer- 

_ method. The seam between the blended images is quite . ence on Computer Vision and Pattern Recognition 

apparent in the image resulting from the simple averaging (CVPR^T),. pages 338-343, San Juan, Puerto Rico, June 

methods FIG. 9B). In addition, while somewhat improved, 1997. 

the blended image of FIG. 9C, still exhibits a readily 35 [RK76] A. Rosenfeld and A. C. Kak. Digital Picture Pro- 

discern able seam at the upper middle part of the composite cessing. Academic Press, New York, N.Y. 1976. 

image. FIG. 9D shows the results of using the weighted [SB96] A. R. Smith and J. F. Blinn. Blue screen matting. In 

pyramid feathering process according to the present inven- Computer Graphics Proceedings, Annual Conference 

tion. As can be seen in this figure, the process results in a Series, pages 259-268, Proc. SIGGRAPH'96 (New 

seamless mosaic. 40 Orleans), August 1996. ACM SIGGRAPH. 
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[Wol90] G. Wolberg. Digital Image Warping. IEEE Com- 
puter Society Press, Los Alamitos, Calif. 1990. 
What is claimed is: 

1. A computer- implemented process for inverse texture 
mapping using a weighted pyramid feathering scheme, 
comprising using a computer to perform the following steps: 

inputting at least two images of a 3D scene; 

setting any pixel of an inputted image located in a cut-out 
region of the image to a zero alpha value; 

identifying corresponding regions of the 3D scene which 
are to be texture mapped in each inputted image; 

computing a 2D perspective transform for each identified 
region of each inputted image, said transform being 
capable of projecting the associated region to pre- 
scribed texture map coordinates; 

warping each identified region from the inputted images 
to the prescribed texture map coordinates using the 2D 
perspective transforms to create plurality of warped 
images; 

creating a weight map for each warped image that speci- 
fies the degree to which each pixel in a particular 
warped image is to contribute to a blended image; 

blending the warped images using the weight maps. 

2. The process of claim 1, wherein prior to the step of 25 
blending the warped images, the process comprises perform- 
ing the steps of: 

determining whether the warped images are in registration 

with one another; 
aligning the warped images whenever it is determined 

they are not in registration. 

3. The process of claim 1, wherein the step of blending the 
warped images comprises the steps of: 

forming an alpha premultiplied image from each warped 
image; 

constructing a band-pass Laplacian pyramid from each 
alpha premultiplied image, wherein each Laplacian 
pyramid comprises a prescribed number of pyramid 
levels; 

constructing a low-pass Gaussian pyramid from each of 
the previously created weight maps associated with 
each warped image, wherein each Gaussian pyramid 
comprises the same number of pyramid levels as each 
Laplacian pyramid; 

computing a new composite band-pass Laplacian pyramid 
using the band-pass Laplacian pyramids and low-pass 
Gaussian pyramids associated with the warped images; 

producing a blended image by expanding and summing 
the pyramid levels of the composite band-pass Lapla- 
cian pyramid; and 

compensating for the alpha premultiplication associated 
with the step of forming the alpha premultiplied image 
of each warped image. 

4. The process of claim 3, wherein the step of computing 
the new composite band-pass Laplacian pyramid comprises 
the steps of: 

for each corresponding pyramid level of the band-pass 
Laplacian pyramids and the low-pass Gaussian 
pyramids, 

multiplying the pixel at each pixel location of each of 
the warped images by a weight factor associated with 
the same pixel location in the weight map associated 
with the warped image to produce a weighted pixel 
value, 

summing the weighted pixel values for each corre- 
sponding pixel location of all the warped image, and 
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dividing the summed weighted pixel value associated 
with a pixel location by the sum of the weight factors 
corresponding with that pixel location in each weight 
map to create a portion of the blended image, except 
whenever the sum of the weight factors for a pixel is 
equal to zero, that pixel is set to zero. 

5. The process of claim 1, wherein the step of forming an 
alpha premultiplied image from each warped image com- 
prises the step of multiplying R, G and B components of 
each warped image pixel, which along with an alpha com- 
ponent define each warped image pixel, by the alpha value 
associated with the particular pixel being multiplied, and 
wherein the alpha premultiplication compensation step com- 
prises dividing R, G, B components of each pixel of the 
blended image, which along with an alpha component define 
each blended image pixel, by the alpha value associated with 
the particular pixel being divided. 

6. The process of claim 5, further comprising, prior to the 
alpha premultiplication compensation step, the steps of: 

respectively clipping the R, G, B and alpha components of 
each pixel in the blended image by first setting any of 
said components having a value of less than zero to 
zero, and secondly setting any of the R, G, and B 
components having a value greater than the alpha value 
of the pixel being clipped to the alpha value. 

7. The process of claim 3, further comprising, following 
the alpha premultiplication compensation step, the steps of: 

comparing each pixel of the blended image to a corre- 
spondingly located pixel of each of the alpha premul- 
tiplied images; 

setting any compared pixel of the blended image to zero 
whenever it coincides with a zero-valued pixel in all of 
the alpha premultiplied images; and 

leaving any compared pixel of the blended image 
unchanged whenever it does not coincide with a zero- 
valued pixel in all of the alpha premultiplied images. 

8. A system for inverse texture mapping using a weighted 
pyramid feathering scheme, comprising: 

a general purpose computing device; 
a computer program comprising program modules 
executable by the computing device, wherein the com- 
puting device is directed by the program modules of the 
computer program to, 
input at least two images of a 3D scene, 
set any pixel of an inputted image located in a cut-out 

region of the image to a zero alpha value, 
identify corresponding regions of the 3D scene which 

are to be texture mapped in each inputted image, 
compute a 2D perspective transform for each identified 
region of each inputted image, said transform being 
capable of projecting the associated region to pre- 
scribed texture map coordinates, 
warp each identified region from the inputted images to 
the prescribed texture map coordinates using the 2D 
perspective transforms to create plurality of warped 
images, 

create a weight map for each warped image that speci- 
fies the degree to which each pixel in a particular 
warped image, is to contribute to a blended image, 
and 

blend the warped images using the weight maps. 

9. The system of claim 8, wherein the computer program 
further comprises program modules, executable prior to the 
execution of the program module for blending the warped 
images, for: 

determining whether the warped images are in registration 
with one another; and 
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aligning the warped images whenever it is determined 
they are not in registration. 

10. The system of claim 8, wherein the program module 
for blending the warped images comprises sub-modules for: 

forming an alpha premultiplied image from each warped 5 
image; 

constructing a band-pass Laplacian pyramid from each 
alpha premultiplied image, wherein each Laplacian 
pyramid comprises a prescribed number of pyramid 
levels; 10 

constructing a low-pass Gaussian pyramid from each of 
the previously created weight maps associated with 
each warped image, wherein each Gaussian pyramid 
comprises the same number of pyramid levels as each 
Laplacian pyramid; 15 

computing a new composite band-pass Laplacian pyramid 
using the band-pass Laplacian pyramids and low-pass 
Gaussian pyramids associated with the warped images; 

producing a blended image by expanding and summing 2 o 
the pyramid levels of the composite band-pass Lapla- 
cian pyramid; and 

compensating for the alpha premultiplication associated 
with the step of forming the alpha premultiplied image 
of each warped image. 25 

11. The system of claim 10, wherein the sub-module for 
computing the new composite band-pass Laplacian pyramid 
comprises sub-modules for: 

at each corresponding pyramid level of the band-pass 
Laplacian pyramids and the low-pass Gaussian 30 
pyramids, 

multiplying the pixel at each pixel location of each of 
the warped images by a weight factor associated with 
the same pixel location in the weight map associated 
with the warped image to produce a weighted pixel 35 
value, 

summing the weighted pixel values for each corre- 
sponding pixel location of all the warped image, and 

dividing the summed weighted pixel value associated 
with a pixel location by the sum of the weight factors 40 
corresponding with that pixel location in each weight 
map to create a portion of the blended image, except 
whenever the sum of the weight factors for a pixel is 
equal to zero, that pixel is set to zero. 

12. The system of claim 8, wherein the sub- module for 45 
forming the alpha premultiplied image from each warped 
image comprises a sub-module for multiplying R, G and B 
components of each warped image pixel, which along with 

an alpha component define each warped image pixel, by the 
alpha value associated with the particular pixel being 50 
multiplied, and wherein the alpha premultiplication com- 
pensation sub-module comprises a sub -module for dividing 
R, G, B components of each pixel of the blended image, 
which along with an alpha component define each blended 
image pixel, by the alpha value associated with the particular 55 
pixel being divided. 

13. The system of claim 12, further comprising a sub- 
module, executable prior to the alpha premultiplication 
compensation sub-module, for respectively clipping the R, 

G, B and alpha components of each pixel in the blended 60 
image by first setting any of said components having a value 
of less than zero to zero, and secondly setting any of the R, 
G, and B components having a value greater than the alpha 
value of the pixel being clipped to the alpha value. 

14. The system of claim 10, further comprising sub- 65 
modules, executable following the alpha premultiplication 
compensation sub-module, for: 
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comparing each pixel of the blended image to a corre- 
spondingly located pixel of each of the alpha premul- 
tiplied images; 

setting any compared pixel of the blended image to zero 
whenever it coincides with a zero-valued pixel in all of 
the alpha premultiplied images; and 

leaving any compared pixel of the blended image 
unchanged whenever it does not coincide with a zero- 
valued pixel in all of the alpha premultiplied images. 

15. A computer-readable memory for causing a computer 
to perform inverse texture mapping using a weighted pyra- 
mid feathering scheme, comprising: 

a computer-readable storage medium; and 
a computer program comprising program modules stored 
in the storage medium, wherein the storage medium is 
so configured by the computer program that it causes 
the computer to, 

input at least two images of a 3D scene, 

set any pixel of an inputted image located in a cut-out 
region of the image to a zero alpha value, 

identify corresponding regions of the 3D scene which 
are to be texture mapped in each inputted image, 

compute a 2D perspective transform for each identified 
region of each inputted image, said transform being 
capable of projecting the associated region to pre- 
scribed texture map coordinates, 

warp each identified region from the inputted images to 
the prescribed texture map coordinates using the 2D 
perspective transforms to create plurality of warped 
images, 

create a weight map for each warped image that speci- 
fies the degree to which each pixel in a particular 
warped image is to contribute to a blended image, 
and 

blend the warped images using the weight maps. 

16. The computer- readable memory of claim 15, wherein 
the computer program further comprises program modules, 
executable prior to the execution of the program module for 
blending the warped images, for: 

determining whether the warped images are in registration 

with one another; and 
aligning the warped images whenever it is determined 

they are not in registration. 

17. The computer-readable memory of claim 15, wherein 
the program module for blending the warped images com- 
prises sub-modules for: 

forming an alpha premultiplied image from each warped 
image; 

constructing a band-pass Laplacian pyramid from each 
alpha premultiplied image, wherein each Laplacian 
pyramid comprises a prescribed number of pyramid 
levels; 

constructing a low-pass Gaussian pyramid from each of 
the previously created weight maps associated with 
each warped image, wherein each Gaussian pyramid 
comprises the same number of pyramid levels as each 
Laplacian pyramid; 

computing a new composite band-pass Laplacian pyramid 
using the band-pass Laplacian pyramids and low-pass 
Gaussian pyramids associated with the warped images; 

producing a blended image by expanding and summing 
the pyramid levels of the composite band-pass Lapla- 
cian pyramid; and 

compensating for the alpha premultiplication associated 
with the step of forming the alpha premultiplied image 
of each warped image. 
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18. The computer-readable memory of claim 17, wherein 
the sub-module for computing the new composite band-pass 
Laplacian pyramid comprises sub- modules for: 

at each corresponding pyramid level of the band-pass 
Laplacian pyramids and the low-pass Gaussian 
pyramids, 

multiplying the pixel at each pixel location of each of 
the warped images by a weight factor associated with 
the same pixel location in the weight map associated 
with the warped image to produce a weighted pixel 
value, 

summing the weighted pixel values for each corre- 
sponding pixel location of all the warped image, and 

dividing the summed weighted pixel value associated 
with a pixel location by the sum of the weight factors 
corresponding with that pixel location in each weight 
map to create a portion of the blended image, except 
whenever the sum of the weight factors for a pixel is 
equal to zero, that pixel is set to zero. 

19. The computer-readable memory of claim 15, wherein 
the sub-module for forming the alpha premultiplied image 
from each warped image comprises a sub-module for mul- 
tiplying R, G and B components of each warped image pixel, 
which along with an alpha component define each warped 
image pixel, by the alpha value associated with the particular 
pixel being multiplied, and wherein the alpha premultipli- 
cation compensation sub- module comprises a sub-module 
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for dividing R, G, B components of each pixel of the blended 
image, which along with an alpha component define each 
blended image pixel, by the alpha value associated with the 
particular pixel being divided. 

20. The computer-readable memory of claim 19, further 
comprising a sub-module, executable prior to the alpha 
premulti plication compensation sub -module, for respec- 
tively clipping the R, G, B and alpha components of each 
pixel in the blended image by first setting any of said 
components having a value of less than zero.to zero, and 
secondly setting any of the R, G, and B components having 
a value greater than the alpha value of the pixel being 
clipped to the alpha value. 

21. The computer-readable memory of claim 17, further 
comprising sub-modules, executable following the alpha 
premultiplication compensation sub-module, for: 

comparing each pixel of the blended image to a corre- 
spondingly located pixel of each of the alpha premul- 
tiplied images; 

setting any compared pixel of the blended image to zero 
whenever it coincides with a zero-valued pixel in all of 
the alpha premultiplied images; and 

leaving any compared pixel of the blended image 
unchanged whenever it does not coincide with a zero- 
valued pixel in all of the alpha premultiplied images. 
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